Optimise dockerfiles, fix ci, release images #444

spwoodcock · 2023-11-08T14:26:39Z

Fixes #439

Updated to multi-stage builds.
Only installed required runtime libs for final underpass image.
Reordered layers for more efficient caching.
Added labels.

Significant image size reductions:

api 1.31GB --> 228MB
underpass 2.5GB --> 1.88GB
ui 1.1GB --> 18.5MB

The underpass image could probably be reduced further by selective usage of boost libraries, instead of libboost-all-dev.

Note

With the images being named ghcr.io/xxx/xxx it would be possible to push them to the underpass repo container registry.

Update 09/11/2023

Added stages to image:
- debug for local testing.
- ci for ci tests.
- prod for production image.
Added workflow for publishing release images.
Set ci_local.sh to use the ci image, with bundled entrypoint.
Added github workflow local testing via act
Made ports user configurable via env vars.
Made image names configurable via env vars.
Note: ci_local uses docker compose V2 (docker compose instead of docker-compose)
- It's the standard tool now bundled with docker install (a re-implementation in Go instead of Python, better in every way).

emi420

The build is working, but some things are failing or missing.

The directory /code changed, and with this, the bootstrap script stopped working for Docker
The API port changed, so the UI is not working now
The source files and the utils folder is missing, so the bootstrap script is not available anymore

Keep in mind that the Docker image is not only for deployment, but also for running Underpass locally. Maybe we can have two different Docker images, one for deployment and one for development/testing/running locally.

docker/underpass-ui.dockerfile

docker/underpass.dockerfile

docker-compose.yml

docker/underpass.dockerfile

spwoodcock · 2023-11-09T01:59:06Z

The standard way to have different environments is to add different stages to the dockerfile.

I usually have a debug stage, and a prod stage. The target can be specified when you build (using docker compose, gh-workflows, etc).

spwoodcock · 2023-11-09T03:55:36Z

Made updates: see in the first post.

Let me know if anything else needs updating 👍

spwoodcock · 2023-11-09T13:21:21Z

Made the final update just now:

Create debug images for local debug stage and prod stage for the actual releases.
Bind mount local code into the containers so you get live updates for both fastapi and cosmos.

I can build and push the images to the hotosm container registry.
The only thing left to do @emi420 is to associate the packages with the repo (only the repo owner can do that):
https://github.com/orgs/hotosm/packages/container/package/underpass
https://github.com/orgs/hotosm/packages/container/package/underpass%2Fapi
https://github.com/orgs/hotosm/packages/container/package/underpass%2Fui
'Link this package to a repository'

spwoodcock · 2023-11-20T23:33:24Z

I will rebase with master to solve conflicts.

emi420

It's mostly working! with docker-compose up -d the UI is not working but if I change the port is working perfectly:

ports:
- "${UI_PORT:-5000}:5000"

CI is failing, possibly because the workflow version and GitHub settings, you know better what changes are needed there.

Thanks @spwoodcock

docker-compose.yml

spwoodcock · 2023-12-01T09:13:44Z

Easiest to merge this to test my theory: the workflow is failing with image reg permissions as it was created before the GITHUB_TOKEN permissions were set correctly 👍

spwoodcock · 2023-12-01T09:21:51Z

That seemed to work 🎉

emi420 reviewed Nov 8, 2023

View reviewed changes

docker/underpass-ui.dockerfile Outdated Show resolved Hide resolved

docker/underpass.dockerfile Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

docker/underpass.dockerfile Outdated Show resolved Hide resolved

spwoodcock changed the title ~~Optimise dockerfiles (multi stage, caching)~~ Optimise dockerfiles, fix ci, release images Nov 9, 2023

spwoodcock mentioned this pull request Nov 9, 2023

Fix CI when using git worktree #103

Closed

spwoodcock added 16 commits November 21, 2023 12:28

build(fix): change port for ui back to 5000

7ce8d09

build: update labels & use /code dir over /opt

945f002

build: add debug and prod stage to underpass dockerfile

1d7505e

build: allow port bind overrides in compose

a6a0b24

ci(release): add workflow to build release images

ee9cd55

build: add ci stage in dockerfile, use for tests

090bba1

build: default to target: debug in local compose

9ed590f

ci: rename release image file --> yml

dd72e02

ci: update ci_local.sh to use compose v2, simplify

7b95f96

ci(tests): use reusable workflow test_compose for build & test

7315bb0

test: add act config for local gh-workflow testing

cb54001

build: add debug stages to dockerfiles for live reload

bc4bd3a

ci: build release imgs to prod docker target

1558e77

build: allow bind mounting code for debug images

883040d

build: rename underpass/server image --> underpass

ed9f68f

build(fix): underpass-ui image change build dir --> /code

cb23f63

spwoodcock temporarily deployed to test November 21, 2023 12:32 — with GitHub Actions Inactive

spwoodcock requested a review from emi420 November 21, 2023 12:33

spwoodcock had a problem deploying to 444/merge November 21, 2023 12:33 — with GitHub Actions Failure

spwoodcock had a problem deploying to test November 21, 2023 12:33 — with GitHub Actions Failure

fix: rebase conflicts for multistage dockerfiles

4291222

spwoodcock temporarily deployed to test November 21, 2023 12:38 — with GitHub Actions Inactive

spwoodcock had a problem deploying to 444/merge November 21, 2023 12:39 — with GitHub Actions Failure

emi420 added this to the Real-time monitoring of digital mapping into the Tasking Manager milestone Nov 30, 2023

emi420 suggested changes Nov 30, 2023

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

emi420 temporarily deployed to test November 30, 2023 07:27 — with GitHub Actions Inactive

emi420 had a problem deploying to 444/merge November 30, 2023 07:27 — with GitHub Actions Failure

emi420 had a problem deploying to test November 30, 2023 07:30 — with GitHub Actions Failure

ci: update workflow versions 1.3.0 --> 1.4.0

356cd33

spwoodcock temporarily deployed to test November 30, 2023 07:34 — with GitHub Actions Inactive

spwoodcock had a problem deploying to 444/merge November 30, 2023 07:34 — with GitHub Actions Failure

spwoodcock had a problem deploying to test November 30, 2023 07:35 — with GitHub Actions Failure

spwoodcock added 2 commits November 30, 2023 12:20

build: fix underpass-ui compose port mapping

b8b4593

ci: update workflows 1.4.0 --> 1.4.1

e217b44

spwoodcock temporarily deployed to test November 30, 2023 12:21 — with GitHub Actions Inactive

spwoodcock had a problem deploying to 444/merge November 30, 2023 12:21 — with GitHub Actions Failure

spwoodcock had a problem deploying to test November 30, 2023 12:24 — with GitHub Actions Failure

spwoodcock temporarily deployed to test November 30, 2023 12:42 — with GitHub Actions Inactive

spwoodcock had a problem deploying to 444/merge November 30, 2023 12:43 — with GitHub Actions Failure

spwoodcock had a problem deploying to test November 30, 2023 12:45 — with GitHub Actions Failure

spwoodcock requested a review from emi420 November 30, 2023 12:49

emi420 assigned spwoodcock Nov 30, 2023

spwoodcock temporarily deployed to test November 30, 2023 14:10 — with GitHub Actions Inactive

spwoodcock had a problem deploying to 444/merge November 30, 2023 14:10 — with GitHub Actions Failure

spwoodcock had a problem deploying to test November 30, 2023 14:13 — with GitHub Actions Failure

ci: allow manual trigger of test workflow

06c131d

spwoodcock temporarily deployed to test December 1, 2023 09:12 — with GitHub Actions Inactive

spwoodcock had a problem deploying to 444/merge December 1, 2023 09:13 — with GitHub Actions Failure

spwoodcock had a problem deploying to test December 1, 2023 09:15 — with GitHub Actions Failure

spwoodcock merged commit 374eff9 into hotosm:master Dec 1, 2023
2 of 4 checks passed

spwoodcock deleted the build/multi-stage-dockerfiles branch December 1, 2023 09:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimise dockerfiles, fix ci, release images #444

Optimise dockerfiles, fix ci, release images #444

spwoodcock commented Nov 8, 2023 •

edited

Loading

emi420 left a comment

spwoodcock commented Nov 9, 2023

spwoodcock commented Nov 9, 2023

spwoodcock commented Nov 9, 2023

spwoodcock commented Nov 20, 2023

emi420 left a comment

spwoodcock commented Dec 1, 2023

spwoodcock commented Dec 1, 2023

Optimise dockerfiles, fix ci, release images #444

Optimise dockerfiles, fix ci, release images #444

Conversation

spwoodcock commented Nov 8, 2023 • edited Loading

Note

Update 09/11/2023

emi420 left a comment

Choose a reason for hiding this comment

spwoodcock commented Nov 9, 2023

spwoodcock commented Nov 9, 2023

spwoodcock commented Nov 9, 2023

spwoodcock commented Nov 20, 2023

emi420 left a comment

Choose a reason for hiding this comment

spwoodcock commented Dec 1, 2023

spwoodcock commented Dec 1, 2023

spwoodcock commented Nov 8, 2023 •

edited

Loading